We provide a more extension discussion for the context of this work. Firstly, when closed-form expressions for the optimizer of a function are unavailable, solving optimization problems requires iterative schemes such as gradient ascent [31]. Their convergence to global extrema is predicated on concavity and the tractability of computing ascent directions. When the objective takes the form of an expected value of a function parameterized by a random variable, stochastic approximations are required [36, 24]. The PG Theorem mentioned above gives a specific form for obtaining ascent directions with respect to a parameterized family of stationary policies via trajectories in a Markov decision process, when the objective is the expected cumulative return [44], which gives rise to the REINFORCE algorithm.

Additional Feedback: Update: thanks for the answer, it helped clarify some points. I think the proposed additions will improve the clarity of the paper. While providing a common theoretical ground for general utilities in RL is not a minor contribution by any means, I would have loved to find a discussion on how to build upon these results. Do authors think their work can be leveraged to develop more efficient algorithm in the context of RL with general utilities, or the intended outcome is a deeper understanding of the setting without particular practical upsides? 2. Where the Variational Policy Gradient approach stands in comparison with other policy optimization methods for (specific) general utilities, e.g.

The paper proposes an unifying view on several interesting problems for the RL community (reward maximization, pure-exploration, risk averse RL). It presents a generic Policy Gradient Theorem and studies the convergence of the corresponding policy gradient ascent, which is an important contribution.

In recent years, reinforcement learning systems with general goals beyond a cumulative sum of rewards have gained traction, such as in constrained problems, exploration, and acting upon prior experiences. In this paper, we consider policy optimization in Markov Decision Problems, where the objective is a general utility function of the state-action occupancy measure, which subsumes several of the aforementioned examples as special cases. As this means that dynamic programming no longer works, we focus on direct policy search. Analogously to the Policy Gradient Theorem \cite{sutton2000policy} available for RL with cumulative rewards, we derive a new Variational Policy Gradient Theorem for RL with general utilities, which establishes that the gradient may be obtained as the solution of a stochastic saddle point problem involving the Fenchel dual of the utility function. We develop a variational Monte Carlo gradient estimation algorithm to compute the policy gradient based on sample paths.